DOCORSHT BESOHE 



EB 121 850 



TH 005 282 



AOTHOS 
TITLE 
P08 DATE 
NOTE 



Carlo Studies. 



BalperiOf Silas 
Design of Monte 
[Apr 76 3 

16p.; Paper presented at the Annual Meeting of the 
American Educational Besearch Association (60th, San 
Francisco, California^ April 19-23, 1976) 



EDBS PEICE HF-$0,83 HC-$1,67 Plus Postage 

DESCBIPTOSS ^Computer Programs; Hypothesis Testing; Besearch 

Methodology; ^Simulation; ^Statistical Analysis; 
Tests of Significance 

IDENTIFIEBS *Monte Carlo Methods 



ABSTRACT 

There are good reasons for the groviii9 popularity of 
Monte Carlo procedures; but vith increasing use comes increasing 
misuse. A variety of exact and approximate alternatives should be 
considered before one chooses to approach a problem vlth Monte Carlo 
methods. Once it has b^ien decided that simulation is desirable, 
consideration should be given to making the study as efficient and 
general as possible. A simpler model or a canonical form can often 
make results more general vhile at the same time reducing the cost of 
the study* (Author) 



***********i(ri(r*****i|e****i(r ***************** 

* Documents acgoired by ERIC include many informal unpublished * 

* materials not available from other sources* EBIC makes every effort * 

* to obtain the best copy available. Nevertheless, items of marginal * 

* reproducibility are often encountered and this affects the quality * 

* of the microfiche and hardcopy reproductions EBIC makes available * 

* via the ERIC Document Reproduction Service (EDHS)» EDHS is not * 

* responsible for the quality of the original document* Reproductions * 

* supplied by EDRS are the best that c^n be f^^de from the original* * 

**************:([ *}tt)fic:St:^j^*iti******i|r***********i|r****i|r** **************** i(r*4«<{tt 



Design of Monte Cnrlo Studies 



O 

CO 

X'" I 

r\j 

i-H 

Q 



Silas Halperin 
Syracuse Unlverssity 



U OEPA«TW€MT OF HEALTH. 
EDUCATION « WELFARE 
MATlOMAL»N*TlTUTEOF 
EDUCATION 

THIS OOCUWENt HAS SEEN REPRO- 
DUCED EXACTLY AS BECElVEO FrOW 
THE PERSON OR ORGAN I lAT ION ORIGIN' 
ATiNOJT POINtSOF VIEWOROPtMlONS 
StATEO DO NOT NECESSARtLV "fPR^" 
SENTOFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLiCV 



00 



Paper presented at SIG/ES symposium on Monte Carlo Methods, 
AERA Annual Meeting, San Francisco, April, 1976 



2-^ 

ERIC 



Abstract 



I'here are good reasons for the growing popularity of 
Monte Carlo procedures; but with Increasing use comes In- 
creasing misuse. A variety of exact and approximate alter^ 
natives should be considered before one chooses to approach 
a problem with Monte Carlo methods* Once It has been decided 
that simulation Is desirable, consideration should be given 
to making the study as efficient and general as possible* 
A simpler model or a canonical form can often make results 
more general while at the sane time reducing the cost of the 
study* 
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Design of MonCe Carlo Studies 



The tide is at the same time too broad and too narrow. It is too broad 
in the sense that Monte Carlo procedures have application Ear beyond research 
in statistics; yet the paper will be constrained to consider only statisti- 
cal applications. It is too narrow in the sense that we will consider not 
only the design of Monte Carlo studies, but also tfr thought process which 
ahould precede the decision to rely on Monte Carlo methods* 

It is clear that no one starts out to design a Monte Carlo study, or at 
least no one should* Any study is the consequence of a series of questions 
which can be answered by sufficient effort aimed in the proper direction. 
In statistics^ that effort has traditionally been directed Cowards finding 
analytic solutions to the problem. Little needs be said about the advantages 
of solving a problem analytically; all recognize the exactness, generalizabiliCy 
and olegance of such results. It Is also weJ.l understood that, in certain 
applications, mathematical analysis either yields results which are too res- 
trictive to be of real use or is sufficiently complex so as to deter solution* 
In either of these cases^ some substitute for analysis must be found. Monte 
Carlo procedure should be viewed as but one of the many possible such substi- 
tutes , 

Once analysis has been eliminated as a means to answer the question of 
Intc^resL, a variety of alternatives should next be considered. One such 
possibility has great potential but is only rarely found in the literature of 
educational statistics: enumeration* When only discrete variables are con- 
sidered, the ''brute force" approach of direct enumeration will often suffice 
as a substitute for analysis. It is not as elegant and will yield generali- 
zability only grudgingly, but it is often capable of giving results which allow 
us to answer particular questions. Illustrations of enumeration will be given 
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in \:ncr sccttonft* 

Emimeration shares one desirable property wllh onalyBts: it is exact. tC 
exact solutions are not feasible, some approximation technique must be 'em- 
ployed* However, approximation techniques refer to a broad set of procedures 
of which Monte Carlo procedures represent but one small part. The choice of 
a particular approximation technique should be made on the basis of its 
accuracy, efficiency and utility. Illustrations will be given of approxima- 
tions other than Monte Carlo. As these are discussed, it will become apparent 
that Monte Carlo techniques have certain advantages which make them attractive 
as supplements to other approximations. The attractiveness of Monte Carlo 
procedures, when combined with their inherent ease of application, is often 
interpreted as license for routine use. It is partly the purpose of this 
paper to dissuade researcners of this attitude. It If further a purpose to 
Introduce, through Illustrations, a variety of ways in which Monte Carlo tech- 
niques can be improved to yield more general results wit!> greater efficiency. 
Before embarking on these endeavors, it is a useful diversion to consider some 
of the history of Monte Carlo methods. 

The Monte Carlo procedure, although based on statistical principles, was 
not motivated by statistical topics* As observed by Hammersley £t Hatidscomb 
(1964): ''The real use of Monte Carlo methods as a research tool stems from 
work on the atomic bomb during the second world war." The name was first uSed 
by Metropolis & Uiam (1949) and the procedure is usually attributed to Ulam, 
von Meumann and Fermi. However, informal use of empirical sampling procedures 
predates these applications by almost a century* During the Civil War, officers 
found a diversion from more serious matters by tossing needles on a board ruled 
with parallel straight lines to infer the^ value of tt . In 1901, Lord Kelvin 
studied Boltzman equations by Monte Carlo methods* In the early part of the 
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twentieth century, Monte Carlo procedures? were used by the British statLsti- 
ciil fichools for their pedagogical value. Gossett (Student^ 1908) was led to 
the distribution of the correlation coefficient partly tlirough the use of 
empirical random sampling* 

Within the past decade, the application of Monte Carlo procedures to 
statistical problems has enjoyed a popularity not possible prior to the advent 
of high speed computers. In a recent survey (Hoaglin nnd Andrews, 1975) it 
was found that roughly one of every five papers published in JASA and 
Blometrika in 1973 contained results from computer simulation* It is clear 
that a technique which was once relegated to second class status is now 
accepted in the most prestigious journals as a legitimate tool of the stat- 
istician* Unfortunately, as the number and variety of legitimate applications 
grows, concomitantly the number of misuses also grows* For example. Games 
(1971) pointed out that portions of a study by Petrinovitch and Hardyck (1969) 
could have been approached analytically rather than by Monte Carlo approxi^ 
mations. This illustrates the need for careful thought prior to selecting 
an approach to a research problem* 

Alternatives to Mathematical Analysis 
Many statisticians, especially those working with the small sample 
characteristics of test ctatistics which have known asymptotic distributions, 
find enumeration a useful alternative to analysis. For example, Odoroff (1970) 
studied the properties of a variety of statistics and estimators used to test 
interaction in RxCx2 contingency tables* He calculated the small sample 
error rates of these tests by enumerating all possible samples for selected 
parameter sets and resorted to Monte Carlo approximations only when his sample 
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size grew beyond the point that enumeration became unwieJdly . Although tlio 
results of enumeration depend upon the selection of parameters, OdorofC was able 
to demonstrate a certain degree of invariance to the selection of a particular 
parameter set. He was thus able to overcome one disadvantage of this non- 
analytic approach, its lack of generalizability, Margolin and Light (1974) 
approached the comparison of three statistics to test homogeneity in contin- 
gency tables in a similar fashion. Thty again demonstrate that enumeration 
can provide solutions to questions concerning tests on categorical data. 

When no exact formulation of the problem is possible, it is necessary 
to resort to approximations. Gosslee and Lucas (1965), studying the proper- 
ties of tests based on the additive sums of squares methods applied to dispro- 
portionate data, found exact distributions available only when the null hy- 
pothesis was true; approximations were necessary to study comparative power 
of the tests. For this reason, they decided to study the approximations under 
the null hypothesis as well as under alternatives so that relative precision 
could be determined. Two types of approximations were employed. The first, 
developed by Box (195A) , used a chi-square distribution with degrees of 
freedom adjusted to be consistent with certain moments of the quadratic form 
under study. The second was Monte Carlo approximation based on 400 replica- 
tions. A comparison indicated that the results of the chi-square approxima- 
tion were closer to the exact results. One might feel this stems from the 
very small number of replications used (the original experiment was performed 
by Gosslee In 1956); however DuPuls (1974), in a similar study, found the 
chi-square approximation to be superior to Monte Carlo results based on 5000 
replications. This should not be interpreted as a total repudiation of Monte 
Carlo procedures. In that same study, DuPuis found a feature of Monte Carlo 
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prn('i>duros wiiich t-oulil ni)t hi* dii|>] 1 c^*1t^*(l hy the t>tht^r 1"*^*^ itui, \i WcIK 

of Init^resL to nLudy the joint cl i^trlhuL Ion of tUv vvirlons l t*HL^; *ih well /ih 
tlieir marginal distrihutionfi, Monte {);irlt) procedures were well suited to ".hia 
purpose and their results were a valuable fiupplenient ti> tliOKc from the chi- 
square? approximation^ 

The Use of Monte Carlo Techniques 
When it becomes clenr that Monte Cnrlo techniques sfiould be used either 
as a supplement to other approaches? or hy themselves, the dt'sign of the experi- 
ment m\jst be carefully planned. Monte Carlo techniques require the total 
specification of the populcitlon and conditions of ^^mplin^. The results may 
di^piM>d upon a variety of such exogenous variable ami, for that reason, these 
variables siiould he varied (or fi^ed) in such a m;intier tliat the results possesss 
some general Iznbility . Studleti relying on Monte Carlo proiedureFi typically 
specify the levels of the various exogenous variables and consider all combin-- 
ationss of them in a completely crossed design. If adiiered to too rigidly, 
this approach can limit the value of the study. 

It is necessary to start with a plan of action and the completely crossed 
layout is usually a reasonable choice. Once into the study it is often the 
case that, due to unforseen intermediate results, the plan becomes inappro- 
priate. There are several types of intermediate results which should dictate 
a change of plans In the middle of a study* We may, for example, have inter- 
est tn the effects of sample size (N) and number of variables (p) on a 
certain statistic, 1 f we find that varying N within some selected range 
has little effect, irrespective of p , we should try values of N otiier 
than those specified by our plan. We must have the flexibility to make Huch 
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revisions, A change of plans is also dictated by excessive predictability, 
tf, on the l>c'i«is of Intermediate outcomes^ the resnJts from sucressivt^ t'etl^^ 
become obvious, those cells should not be filled only for the sake of completing 
the crossed layout, Monte Carlo procedures tend to be expensive and our re- 
sources should be allocated to situations which provide maximum information. 
In the same vein, the "cheapest*^ cells (e,g,, those with smallest N, smallest 
p ) should be filled first to avoid possible overspending on non-useful cases. 

It is not reasonable to attempt to dictate how many replications are 
needed to make Monte Carlo results sufficiently stable. It Is reasonable to 
sugge,^t th,it question be given some priority at Llie plnnnfrif^ staf^e. There 
will be Jitudies where minimum differences may be of v;ilue nnd the number of 
replications must be huge. In other situations, a fnlrly fsmall number of 
replications (e,g,5 400 in the Gosslee (1956) study) may suffice. In any case, 
we might be well advised to adopt a Tukey philosophy toward Monte Carlo results 
and Ret as much detail as possible output. If more replications are found to 
be needed, the extra output may allow us to do more running and combine runs 
by hand. 

Because of the general availability of random number generators and the 
relative ease of approaching problems with Monte Carlo techniques, we can 
readily be lulled into a philosophy of letting the computer do our thinking for 
us. The careful planning of a Monte Carlo study is important not only to insure 
its generalizability, but also to limit its expense. At first glance, expense 
and generalizability appear to be competing goals. That need not be the case 
however* Certain general ideas exist in the Monte Carlo and statistics liter- 
ature which allow us to increase the generalizability of results while at the 
same time limiting tlie costs* These teehniques falj undt^r the general ephithet 
of variance reduction, 
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Handscomb (1969) suggests that we can regard viirlance and work as 

mutually convertible, and so are led to define ef f ictency with which a simu- 
lation estimates a parameter by 

efficiency = l/(varlance x work)* 

A technique Is called variance-reducing In the Monte Carlo sense If It In-- 
creases the efficiency; that Is If It reduces the variance proportionately more 
than It Increases the work Involved/' For some reason, simulation studies In 
areas other than statistics have taken fuller advantage of variance reduction 
techniques than have statistical studies. 

Let us begin an Investigation of variance reduction techniques with a 
simple example, Prowda (1975) applied a nested analysis of variance model to 
Item sampling In test construction. He wished to generate data according to 
the normal ogive model but recognized the expense of such generation. Substi- 
tuting the logistic model for the normal ogive model gave comparable results 
with less computer time and thus qualifies as a variance reduction technique. 

An even more attractive variance reduction technique was employed by 
Marks and Dunn (1974) and Curlette (1975) in studies of classification proce- 
dures. As with many multivariate models, classification using discriminant 
functions is invariant to linear transformations. For this reason, studies 
such as these can use a canonical form to good advantage. It can be shown 
that, in a two group case, the vector variables I'^l^ ^2"^— 2*^2^ 

can be linearly transformed into the vector variables Zj-(£ » ^) Z2-^^»^^» 
where A is a diagonal matrix. This is accomplished through the use of a 
theorem by Etao (1973, p. 41) which establishes a transformation matrix T such 
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that T'j:,T - I and T'L^T - rt 



With T d-rinc'd l>y 



T 




wherO 





and P^l>^ = 



I, 



our canon- 



ical vector variables are slm,*)ly defined as 



rhore arc sovt^ral important reasons For prefGrrinK to work with these 
Ccin(>[iical vector varlablog* First, they are gonornted with less t^rfort than 
tlio original variables* If we wtt;li to i;encrate samples from multivariate* 
norm;i]s, the elements of the vector ^ ^re independent :ind ean be ^^enerated 
Individually* Second, selection of and A defines an infinite equivalence 
elasrt oC pairs of populations, each pair possessing the identical canonit^al 
form* This fact appears to have been first recognized hy Cur let te (1975) and 
allows us to generalize the results from one canonical selection to an infinite 
numher oC equivalent situations, Thlri ]>rovides an excellent Illustration of 
Slow :\ variance reduction technique ran improve the" jf;eneral i 7,ab il i ty of our 
results while simultaneously reducing; tlie amount of work rer|ulred. 

Another variance reduct ion technique which lias )t>reci t poLt^nt ial but ■ re- 
maln^^ relatively obscure utiliJtes a proi:edure developed by Oiie) I and Peivp«i)n 
(1966)- In their paper, they develop a quick procedure For ^eneratinj; sample 
covariance matrices from a multivariate normal distribution. The advantaj^e of 
their procedure over that suggested by Kaiser and Dickman (1962) stems from 
eliminating the need to generate score vectors. This is critical when sample 
size becomes large* 

The direct generation scheme relies on the ability to generate chl-square 
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v:iri;it)les< Odell and t'eiveson sugj;est .in algorithm to K<-^ner:itci chl-squnrt* 
variables, but the method is approximate and, accordinj^ to Zelen and Sevoro 
(196'?'*) is best for lar^e degrees of freedom. It might be wirie to rei;arL to 
this nppr^xlmation when df are Jarge, but use tlie exponential generation 
^^rhemt^ described by Knuth (1969, pp, 113-115) for smaller df. 

Direct generation of covariance matrices could be of real utility for a 
wide t lass of problems; often it is possible to calculate everything of inter- 
est from the covariance ni;itrix without ever requiring the score vectors. If 
ct^ntroids are addltionnlly required, wc can easily generate them separately, 
fiiiU'e nieanM are Lndopcindent of covririances in saiiiplefi from normal popul;it ionfi . 
Use of thisJ scheme could improve tlie efficiency of studies concerned witfi mul- 
tii>le regression on random predictors, at; well as a wide class of mult ivariai-e 
stud ion . 

Tfie three variance reduction techniques given above are meant to be 
illustrations and should not be viewed as exhaustive. Given a specific prob- 
lem, it is often possible to use ideas such as the three presented to improve 
the study , whil e siniul taneously reducing computer t ime . However, variance 
reduction is not the only means to iniprove Monte Carlo a()plications. One final 
example will be given to ilJustrate how Monte Carlo procedures may be u&ied to 
supplement an analytic approach. 

In an article by Gleason and Halperin (1973), r . odel for data fro^ii a 
round-robin experinient is developed. Under certain conditions, tests of hy- 
potheses about the parameters could be derived analytically. Under more real- 
istic conditions, however, exact tests which are free of nuisance parameters 
could not be developed for certain hypotheses* For this reason, a number of 
approximate tests of the quasi-F variety were suggested. In an attempt to 
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dXsrover which approximate test wns to be preferred, :x rtntfill Mtmtc Carlo 
experiment was performed* Since the M'Jnte Oarlo procedure rcMiuired totnl 
specification of all parameters, the exacc test which depended upon nuisance 



various quasi-F statistics* These compari!=!ons provided an extra criterion 

for deciding which approximate statistic to recommend* Here, the Monte Carlo 

* 

procedure allowed approximate tests, which could he rnlculated on real data, 
to hv compared with an exact test which could ^ot be calculated unless param 
(*torH were known* Since pnrametervS are unknown in actual nppLlcatlon^ the 
ux;u:t test was of no l>ractica 1 vnlu<?» hut possessed h l^n * f it'*;n t thc()rcLj('aI 
vnlut* within the confines of the Monte C;irlo experiment. IL i« rcntionahlc 
to believe that this idea is applicable to a wide varltity of other vSituaL loiia 
when* nuisance parameters deter development of exact tcst«* 

Summary 

IVo major themes ran through this paper; 



I. Every attempt should he made to fiolve our problems 
analytically* When tliLs attempt (*learly is not pro- 
ductive, a range of niLernatives should ht* ronsfdort^d* 
We should not view Monte Carlo techni(|U(*K m(*rely ;is a 
crutch* 

2* When Monte Carlo techniques are clearly fndirated^ wt* 
should make every effort to use such techniques crea^ 
tively* Our literature abounds with methods which ean 
be used to our advantage and, as statisticians, we 
should be in the forefront of such application* 



The illustrations given in this papfer are not original; they all reside in 
our literature. It is interesting that mcsi technlquevS meant to improve the 



parameters could be calculated and used as a ''benchmark** to compare with the 
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Iho }\i^no.rtxi art of rt7in|iut<.*r Rimtilallon :iri; sLatisLlrvil In ii.iiurt' yoi 
st;it r tit Iclnns seldom avail tlu»mfielve« of thcf^c procedurt*!^* 'tUlt^ may bv 
because our problems are so readily congruent to a brute force Monte Cnrlo 
approach that we fall tc appreciate refinements which are easily made* It 
Is lioped that this discussion will alert those statisticians who do use 
Montt.' Carlo techniques to some new possibilities and that we will all be more 
SL^nsitlve In the future to the possible applications of our own discipline* 
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